Skip to content

use ccalendar instead of np_datetime #21549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 26, 2018
Merged

Conversation

jbrockmendel
Copy link
Member

Do Not Merge

AFAICT the np_datetime.c versions of these functions are noticeably more performant than the cython versions. I could use help verifying this observation since asv doesn't work well for me.

Best guess is that lookups in np_datetime.c's days_per_month_table[2][12] are more efficient than lookups in ccalendar.days_per_month_array, as there does not appear to be a way to instantiate the C data structure in cython (at least not as a module-global).

@jreback
Copy link
Contributor

jreback commented Jun 20, 2018

what does the generated code look like? (for that function)

@@ -437,12 +437,6 @@ class BaseOffset(_BaseOffset):
# ----------------------------------------------------------------------
# RelativeDelta Arithmetic

@cython.wraparound(False)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use days_per_month_array in ccalendar.pyx?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use days_per_month_array in ccalendar.pyx?

We could import it from the C file, but we can't define it directly in cython. Since part of the point of ccalendar was to move to pure-cython, I'm not sure we want to re-introduce the C dependency. Or if we do, we might as well keep the entire implementation in the C file.

Cython 0.28 supposedly allows for "verbatim C" in the form:

cdef extern from *:
     """
     foo;
     bar;
     """

which could be useful for small snipped. But when I've tried this I get compile-time errors.

@jbrockmendel
Copy link
Member Author

what does the generated code look like? (for that function)

After stripping out a bunch of comments:

static CYTHON_INLINE __pyx_t_5numpy_int32_t __pyx_f_6pandas_5_libs_6tslibs_9ccalendar_get_days_in_month(int __pyx_v_year, Py_ssize_t __pyx_v_month, CYTHON_UNUSED int __pyx_skip_dispatch) {
  __pyx_t_5numpy_int32_t __pyx_r;

  __pyx_r = (__pyx_v_6pandas_5_libs_6tslibs_9ccalendar_days_per_month_array[(((12 * __pyx_f_6pandas_5_libs_6tslibs_9ccalendar_is_leapyear(__pyx_v_year)) + __pyx_v_month) - 1)]);
  goto __pyx_L0;

  /* function exit code */
  __pyx_L0:;
  return __pyx_r;
}```

@gfyoung gfyoung added Experimental Performance Memory or execution speed performance and removed Experimental labels Jun 21, 2018
@chris-b1
Copy link
Contributor

I think this will all get inlined so performance shouldn't differ anyways, but FYI you can equivalently define/import days_per_month_array if you wanted to.

# ccalender.pyx
cdef int* days_per_month_array

#tslib.pyx
from ccalender cimport days_per_month_array

@jbrockmendel
Copy link
Member Author

@chris-b1 I'm not sure we're on the same page as to the goal/problem.

  • Seemingly-equivalent implementations in C appear to continue outperforming those in cython
  • I think it has to do with the type of array that is used in the C file that can't be constructed as a global in the cython file. (see also comment above regarding cython's new "verbatim C" feature)
  • I've been an advocate for reducing C dependencies and implementing things directly in cython; if there are unavoidable performance implications, I may need to rethink that position.
    • not having cython boilerplate leads (I think) to smaller .so files and (I expect) quicker compilation. Not sure how much either of those matter.
  • Profiling these functions is tricky, so all of this comes with a grain of salt

@chris-b1
Copy link
Contributor

Thanks for the color, I was attempting to say that I think this part is false, or at least seemed to work with what I showed above

the type of array that is used in the C file that can't be constructed as a global in the cython file. (see also comment above regarding cython's new "verbatim C" feature)

Interesting (surprising) that performance is different, I will look closer

@chris-b1
Copy link
Contributor

At least with days_in_month getting similar performance both ways

# master
big_ts = pd.date_range('1999-01-01', periods=1_000_000, freq='H')
%timeit big_ts.daysinmonth
91.6 ms ± 11.7 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# patch
big_ts = pd.date_range('1999-01-01', periods=1_000_000, freq='H')
%timeit big_ts.daysinmonth
94.7 ms ± 6.98 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

patch

diff --git a/pandas/_libs/tslibs/fields.pyx b/pandas/_libs/tslibs/fields.pyx
index ccf67e765..be53dfe8b 100644
--- a/pandas/_libs/tslibs/fields.pyx
+++ b/pandas/_libs/tslibs/fields.pyx
@@ -14,13 +14,19 @@ from numpy cimport ndarray, int64_t, int32_t, int8_t
 cnp.import_array()

 from ccalendar import get_locale_names, MONTHS_FULL, DAYS_FULL
-from ccalendar cimport (get_days_in_month, is_leapyear, dayofweek,
+from ccalendar cimport (is_leapyear, dayofweek,
                         get_week_of_year, get_day_of_year)
 from np_datetime cimport (pandas_datetimestruct, pandas_timedeltastruct,
+                          days_per_month_table,
                           dt64_to_dtstruct, td64_to_tdstruct)
 from nattype cimport NPY_NAT


+@cython.wraparound(False)
+@cython.boundscheck(False)
+cdef inline int get_days_in_month(int year, int month) nogil:
+    return days_per_month_table[is_leapyear(year)][month - 1]
+
 def get_time_micros(ndarray[int64_t] dtindex):
     """
     Return the number of microseconds in the time component of a

@codecov
Copy link

codecov bot commented Jun 24, 2018

Codecov Report

Merging #21549 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master   #21549   +/-   ##
=======================================
  Coverage    91.9%    91.9%           
=======================================
  Files         153      153           
  Lines       49547    49547           
=======================================
  Hits        45537    45537           
  Misses       4010     4010
Flag Coverage Δ
#multiple 90.3% <ø> (ø) ⬆️
#single 41.78% <ø> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1aa08c...f986a5c. Read the comment docs.

@jreback
Copy link
Contributor

jreback commented Jun 25, 2018

@jbrockmendel status of this? (as the DO NOT MERGE at the top is clear!)

@jbrockmendel
Copy link
Member Author

If I understand @chris-b1 correctly then the perf discrepancy may not be an issue, in which case this is a clear (albeit small) improvement, so merge is OK.

@jreback jreback added this to the 0.24.0 milestone Jun 26, 2018
@jreback jreback added the Clean label Jun 26, 2018
@jreback jreback merged commit ebe480a into pandas-dev:master Jun 26, 2018
@jreback
Copy link
Contributor

jreback commented Jun 26, 2018

thanks @jbrockmendel

@jbrockmendel jbrockmendel deleted the use_ccal branch July 1, 2018 01:27
Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Clean Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants